STA141A Final Project
Hannah Wen - 920776210
2024-03-18
Abstract
The brain is one of the most complex organs which hoards tons of information on how we make our decisions. This holds true for not only humans, but mice. The 2019 Steinmetz study on the neural activities of mice during a visual task provide an in-depth look of the ways aspects such as brain areas and neural spikes interact with each other to form a decision. This data is utilized in this project to perform exploratory data analysis to compile all of the data into one structure to be used to train and build a predictive model which will predict whether a mouse turned the wheel in the correct manner using the data provided as well as newly engineered features. This project found that the averaged time bins of each trial and the average number of spikes of the most active neurons proved to have heterogeneous patterns across trials and sessions and were transformed into additional variables. During this exploratory data analysis, it was also noted that behavior varied depending on the type of correct expected turn, resulting in 3 different models being made as the final model. Logistic regression, ridge regression, and random forest models were tested. The final model consists of 1 ridge regression model, 1 random forest model, and 1 random forest model with 10-fold cross validation.
Section 1: Introduction
In 2019, Nicholas A. Steinmetz, Peter Zatka-Haas, Matteo Carandini, and Kenneth D. Harris conducted a study which examined the performance of a group of mice in performing a task which involved turning a wheel to indicate which of the left or right visual stimulus had a larger contrast. The correct expected wheel turn depended on which side had a larger contrast. When the contrasts were equal, the wheel is expected to be held still. There were also random trials in which the contrasts were present but equal, resulting in the correct answer being random. The brain activity of the mice was also recorded after a surgical procedure to attach a steel head plate and recording chamber to monitor brain activity. This study aims to reveal the possible patterns behind brain activity when making decisions on a visual discriminatory task. Whether the mouse turn the wheel correctly or not was noted by the feedback type which either was a -1 for failure or 1 for success. Mice were given rewards if they turned the wheel correctly. The purpose of this project is to examine the information given which may help accurately predict whether a specific trial of a mouse led to a failure or success. Originally this experiment collected the information on the performance of 10 mice over 39 sessions. For the purpose of this project, the data was reduced to 18 sessions covering the performance of 4 different mice.
Section 2: Exploratory Analysis
Each session consists of 8 lists. Within those lists, there are two additional lists named “spks” and “time” which track the spikes of neurons in the visual cortex within the time bins in the list “time”. In “spks”, there are matrices with the number of rows representing the number of neurons and the columns representing the number of time bins. A 1 indicates a neuron fire while a 0 represents no activity. There are 40 times bins and the number of neurons depends on the session. The other 6 lists consist of lists of the mouse name, date, left contrast, right contrast, time stamps, and brain area for each neuron. Each entry in the list represents the information for one trial within the session. The feedback type represents whether the mouse turned the wheel in the expected manner or not. A 1 indicates a success and a -1 indicates failure. The contrasts represent the contrast of the stimulus on the right and left side. The time stamps are the centers for the 40 time bins among each trial.
Summarizing over all sessions, there are 16,305 trials covering the work of mice Cori, Forssmann, Hench, and Lederberg. The overall success rate is approximately 70.74%.
| mouse_name | date_exp | n_brain_area | n_neurons | n_trials | success_rate |
|---|---|---|---|---|---|
| Cori | 2016-12-14 | 8 | 734 | 114 | 0.6052632 |
| Cori | 2016-12-17 | 5 | 1070 | 251 | 0.6334661 |
| Cori | 2016-12-18 | 11 | 619 | 228 | 0.6622807 |
| Forssmann | 2017-11-01 | 11 | 1769 | 249 | 0.6666667 |
| Forssmann | 2017-11-02 | 10 | 1077 | 254 | 0.6614173 |
| Forssmann | 2017-11-04 | 5 | 1169 | 290 | 0.7413793 |
| Forssmann | 2017-11-05 | 8 | 584 | 252 | 0.6706349 |
| Hench | 2017-06-15 | 15 | 1157 | 250 | 0.6440000 |
| Hench | 2017-06-16 | 12 | 788 | 372 | 0.6854839 |
| Hench | 2017-06-17 | 13 | 1172 | 447 | 0.6196868 |
| Hench | 2017-06-18 | 6 | 857 | 342 | 0.7953216 |
| Lederberg | 2017-12-05 | 12 | 698 | 340 | 0.7382353 |
| Lederberg | 2017-12-06 | 15 | 983 | 300 | 0.7966667 |
| Lederberg | 2017-12-07 | 10 | 756 | 268 | 0.6940299 |
| Lederberg | 2017-12-08 | 8 | 743 | 404 | 0.7648515 |
| Lederberg | 2017-12-09 | 6 | 474 | 280 | 0.7178571 |
| Lederberg | 2017-12-10 | 6 | 565 | 224 | 0.8303571 |
| Lederberg | 2017-12-11 | 10 | 1090 | 216 | 0.8055556 |
Figure 2.1: Table of the generalized sessions.
Figure 2.2: Barchart of successes and failures for the sessions of Mouse Cori.
Above are the graphs of successes and failures among the 3 sessions for mouse Cori. For sessions 1-3, the success rates are 60.5%, 63.3%, and 66.2%, respectively. The success rates slowly increase with each session differing by only a few percent between each rate. Sessions 2 and 3 also have considerably higher number of trials than session 1. Session 1 has 114 trials while Session 2 and 3 have 251 and 228 trials, respectively.
Figure 2.3: Barchart of successes and failures for the sessions of Mouse Forssmann.
Sessions 4-7 cover mouse Forssmann’s performance, with success rates of 66.7%, 74.1%, 67.1%, and 64.4%, respectively. The success rate has no clear pattern with the progression of sessions, with success rate peaking during session 5 then going back down. Forssmann has a more consistent number of trials for each session, ranging from 249-290 trials per session. Session 6 has 290 trials while the rest of Forssmann’s lie very close to 250. However, there is no implication on the number of trials and success rate for mouse Forssmann.
Figure 2.4: Barchart of successes and failures for the sessions of Mouse Hench.
Mouse Hench’s performance is covered by sessions 8-11. Hench’s success rates are 64.4%, 68.5%, 62.0%, and 79.5% for sessions 8-11, respectively. There is more variation among the success rates over the 4 sessions, as well as the number of trials. Session 8-11 have 250, 372, 447, and 342 trials, respectively. There is also no distinct pattern between the number of trials and success rates for mouse Hench.
Figure 2.5: Barchart of successes and failures for the sessions of Mouse Lederberg.
Lederberg has fairly high success rates across the last 6 sessions. For sessions 12-18, Lederberg’s success rates are 73.8%, 79.7%, 69.4%, 76.5%, 71.8%, 82.0%, and 80.6%, respectively. The number of trials are 340, 300, 268, 404, 280, 224, and 216, respectively. Over the sessions, Lederberg’s performance does not seem to increase or decrease in a clear pattern.
In conclusion, the success rates for each session for each mouse do not have distinct relationships with the progression of sessions or number of trials. However, it is clear the the performance differs wildly for each mouse, which will be considered while further investigating performance. To further examine the mouse performance for each session, the number of brain areas will be explored next.
Figure 2.6: Graph of number of brain areas plotted against success rate for each mouse.
This plot depicts the relationship between the number of brain areas per session and the associated success rate for the session. Each of the 4 colors of the points represent which mouse it represents. As observed, there is no clear pattern between the two. However, similar success rates for each mouse are observed. Lederberg’s points typically lie on the upper half of the y-axis, with success rates ranging from roughly 70-85%. Hench has points varying wildly on the y-axis, going from as high as nearly 80% to as low as 62%. It also is observed that Hench has a lower success rate when the number of brain areas increases. Forssmann’s success rate varies less, lying between 65-75%, and also decreases as the number of brain areas increases. Cori has the lowest points overall, ranging between 60-67.5%. Cori’s success rate does not appear to have a relationship with the number of brain areas.
It appears that there is no general pattern between the number of brain areas per session and success rate but for each mouse, there may be evidence for further investigation of the brain areas.
Figure 2.7: Barchart of frequency of each brain area across all sessions.
Here is the distribution of the frequency of each area in the brain. Most of the brain areas are only recorded in one session. However, there are several areas which occur in multiple sessions such as ZI and ACA. Only a couple brain areas occur in over 10 different sessions: root and CA1. The most popular brain area is root, with 16 appearances. This demonstrates that certain areas are more present than others. This suggests the possibility of a brain area having a relationship with the success rate. Because every session has a different number of distinct brain areas, this relationship will be examined by taking the three most popular brain areas for each session.
| mouse_name | date_exp | n_brain_area | n_neurons | n_trials | success_rate | area_1 | area_2 | area_3 |
|---|---|---|---|---|---|---|---|---|
| Cori | 2016-12-14 | 8 | 734 | 114 | 0.6052632 | VISp | LS | MOs |
| Cori | 2016-12-17 | 5 | 1070 | 251 | 0.6334661 | VISpm | VISl | POST |
| Cori | 2016-12-18 | 11 | 619 | 228 | 0.6622807 | MG | VISam | VISp |
| Forssmann | 2017-11-01 | 11 | 1769 | 249 | 0.6666667 | LSr | ACA | TH |
| Forssmann | 2017-11-02 | 10 | 1077 | 254 | 0.6614173 | root | OLF | SUB |
| Forssmann | 2017-11-04 | 5 | 1169 | 290 | 0.7413793 | root | TH | AUD |
| Forssmann | 2017-11-05 | 8 | 584 | 252 | 0.6706349 | CA3 | VPL | root |
| Hench | 2017-06-15 | 15 | 1157 | 250 | 0.6440000 | PO | ILA | PL |
| Hench | 2017-06-16 | 12 | 788 | 372 | 0.6854839 | ORBm | VISl | CA1 |
| Hench | 2017-06-17 | 13 | 1172 | 447 | 0.6196868 | MB | POL | VISrl |
| Hench | 2017-06-18 | 6 | 857 | 342 | 0.7953216 | MOp | CP | LSc |
| Lederberg | 2017-12-05 | 12 | 698 | 340 | 0.7382353 | MD | SUB | root |
| Lederberg | 2017-12-06 | 15 | 983 | 300 | 0.7966667 | PL | root | LGd |
| Lederberg | 2017-12-07 | 10 | 756 | 268 | 0.6940299 | MOs | ORB | CA1 |
| Lederberg | 2017-12-08 | 8 | 743 | 404 | 0.7648515 | VPM | GPe | BLA |
| Lederberg | 2017-12-09 | 6 | 474 | 280 | 0.7178571 | TH | SSs | MB |
| Lederberg | 2017-12-10 | 6 | 565 | 224 | 0.8303571 | root | VPM | VPL |
| Lederberg | 2017-12-11 | 10 | 1090 | 216 | 0.8055556 | TH | ZI | CP |
Figure 2.8: Table of the origin brain areas of the top 3 most active neurons for each session.
By looking at the top 3 brain areas for each session, it is observed that root is the most common top brain area by being a top 3 brain area for 6 different sessions for all mice except Cori. TH is observed 4 times, in the sessions for Forssmann and Lederberg. Most brain areas only appear once or twice.
| area | mean_rate |
|---|---|
| root | 0.7398 |
| TH | 0.7329 |
| CA1 | 0.6898 |
| CP | 0.8004 |
| MB | 0.6688 |
| MOs | 0.6496 |
| PL | 0.7203 |
| SUB | 0.6998 |
| VISl | 0.6595 |
| VISp | 0.6338 |
| VPL | 0.7505 |
| VPM | 0.7976 |
Figure 2.9: Table of the average success rates of each origin brain area of the top neurons.
Here is a table of the mean success rate for the brain areas which appeared more than once. The average rate for CP, which occurs once in Hench and Lederberg each, is the highest at 80.04%. The average rate for VISp, which occurs twice in only Cori’s sessions, is the lowest at 63.38%. These rates can be explained by the fact that the associated top brain areas specifically occurred in sessions with a lower success rate. However, this does indicate that the top brain area is dependent on the session, and suggests that there are top brain areas depending on the trial.
To look into the structure of trials, the data structure needs to be standardized across all trials for the data to be explored. A tibble is created with each entry representing information on each trial. Each observation consists of the session number, the mouse name, the date, the trial number within the session, the start time, the left and right contrast, and the feedback type. New features were also added to further examine possible relationships between variables. Turn expectation, which is a string representing the correct expected turn is created through calculating the difference between the left and right contrasts. The names of the top 3 brain areas for the top 3 neurons each trial are recorded as well as their associated average number of spikes. The first, second, and third top brain areas are ordered in decreasing order of how high their firing rates are. The number of distinct brain areas, number of spikes, average number of spikes, number of neurons, and average spikes per neuron are also recorded.
| session_num | mouse_name | date_exp | trial_n | start_time | contrast_left | contrast_right | contrast_diff | turn_expectation | feedback | n_brain_area | n_spikes | avg_n_spikes | n_neurons | avg_spikes_per_neuron | top_area_1 | top_area_2 | top_area_3 | avg_spike_area1 | avg_spike_area2 | avg_spike_area3 |
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| 1 | Cori | 2016-12-14 | 1 | 71.21 | 0.0 | 0.5 | -0.5 | left | 1 | 8 | 1161 | 29.02 | 734 | 1.58 | ACA | VISp | VISp | 0.60 | 0.42 | 0.42 |
| 1 | Cori | 2016-12-14 | 2 | 81.24 | 0.0 | 0.0 | 0.0 | still | 1 | 8 | 963 | 24.08 | 734 | 1.31 | CA3 | LS | VISp | 0.62 | 0.52 | 0.42 |
| 1 | Cori | 2016-12-14 | 3 | 86.81 | 0.5 | 1.0 | -0.5 | left | -1 | 8 | 1354 | 33.85 | 734 | 1.84 | ACA | VISp | LS | 0.78 | 0.55 | 0.52 |
| 1 | Cori | 2016-12-14 | 4 | 95.99 | 0.0 | 0.0 | 0.0 | still | -1 | 8 | 1014 | 25.35 | 734 | 1.38 | CA3 | SUB | VISp | 0.75 | 0.62 | 0.52 |
| 1 | Cori | 2016-12-14 | 5 | 99.56 | 0.0 | 0.0 | 0.0 | still | -1 | 8 | 1046 | 26.15 | 734 | 1.43 | SUB | CA3 | VISp | 0.92 | 0.58 | 0.48 |
Figure 2.10: First 5 rows of expanded dataframe depicting each trial.
|
|
Figure 2.11: Frequency and proportion table of the origin brain area of the most active neuron and feedback type for mouse Cori.
There are 15 distinct areas which are origin for the top firing neuron for 593 trials. MRN appears to be the most popular top brain area for mouse Cori by appearing as the top brain area for 205 trials. Approximately 68.78% of the trials where MRN is the top brain area are successful. The frequency of the top area ranges from 1-205, which makes it harder to draw conclusions on the type of top brain area and success rate. However, CA1 and CA3 both have similar frequencies of 59 and 55, respectively, but CA1’s trials success rate is 59.32% while CA3’s success rate is 45.45%. This is considerable difference which indicates that the top brain areas for each trial may be a valuable feature.
|
|
Figure 2.12: Frequency and proportion table of the the second most active neuron’s brain area and feedback type for mouse Cori
For the second highest firing neuron, the distinct areas appear to be the same with the inclusion of area NB, which only occurs once. This indicates that the most active neurons will tend to come from the same areas. Additionally, the success rates for each area are similar to those of the same area for the top firing neuron. For example, CA1’s success rate is 64.15% which is significantly higher than CA3’s success rate of 50%. MRN remains the most popular brain area origin, with a success rate of 66.7%, very similar to its success rate when it represents the top firing neuron.
|
|
Figure 2.13: Frequency and proportion table of the the 3rd most active neuron’s brain area and feedback type for mouse Cori.
Looking into the frequency and proportion table for brain area origins of the third most active neuron for mouse Cori, it is observed that the number of distinct areas has increased to 17, including the areas MOs, which occurs only once similar to the area NB which was a new addition when moving from the top firing neuron to the second most active neuron. The behavior for certain areas have change. For example, CA1’s success rate is 70% which is very similar to CA3’s success rate of 68.8%. Additionally, POST has a similar frequency of 48 to the POST origin area for the first and second most active neurons, but has a very similar success rate of 64.6% to the POST success rate of 64.3% for the 1st most active neurons while the POST success rate for the second most active neurons is exactly 50%. This demonstrates that the ranking of activeness for the brain origin may have a relationship with the success rate.
Figure 2.14: Violin plots for the average number of spikes of the top neuron for each mouse depending on feedback type.
The averages range from approximately 0.25 to 3.5. It appears that there are not many discrepancies between the mice or feedback type. The ranges for successes appear to be slightly wider than the range for fails across all mice. The distributions are very similar within each mouse. However, it appears that for Cori and Hench, the widest region of the plot for failures is lower than the widest region for successes, indicating that most points are typically located at a slightly lower average number of spikes for failures. For Forssmann, the opposite holds true. For Lederberg, failures have a larger concentration of points but at the same place where the largest concentration of points for successes are. Lederberg has the highest range of average spikes while Cori has the smallest.
Figure 2.15: Violin plots for the average number of spikes for the 2nd most active neurons for each mouse according to feedback type.
The averages range from approximately 0.175 to 2, which is a lower range than the one for the 1st most active neuron. Similar to the first set of violin plots, within each mouse, the distributions are very similar. It is observed that Forssmann has the largest range and Cori has the smallest. Cori’s distribution for failures has a larger concentration of points at the bottom and is slightly different shaped from the one for successes. In this visual, it is also observed that the success distributions have wider ranges than the negative distributions for each mouse. Even Hench, who did not show this behavior for the most active neuron.
Figure 2.16: Violin plots for the average number of spikes for the 3rd most active neuron for each mouse depending on feedback type.
The average ranges from approximately 0.2 to 1.6, which is the lowest and smallest distribution of average spikes among the top 3 neurons. The distributions within each mouse are also similar to each other like the previous plots for the top 2 neurons, with the same observation that certain distributions for failures have their widest concentrations of points at lower averages than the distributions for successes. This pattern is observed for mice Cori and Hench while mice Forssmann and Lederberg have their widest concentrations at very similar values for both failures and successes.
Through observing the violin plots for the average number of spikes for the 3 most active neurons, it is concluded that this feature may not be very valuable to predicting the feedback type due to the similarities of the distributions for each feedback type among the different mice. However, there can be slight differences between the distributions which vary depending on the mouse and neuron activity ranking.
To more rigorously test whether the origin area and average number of spikes for the most active neurons affect the success rate, a two-way anova test will be performed on the origin area of the most active neuron and its average number of spikes.
## Df Sum Sq Mean Sq F value Pr(>F)
## top_area_1 55 128 2.321 2.895 7.91e-12 ***
## avg_spike_area1 1 5 4.856 6.056 0.01389 *
## top_area_1:avg_spike_area1 52 64 1.239 1.545 0.00742 **
## Residuals 4972 3987 0.802
## ---
## Signif. codes: 0 '***' 0.001 '**' 0.01 '*' 0.05 '.' 0.1 ' ' 1Figure 2.17: 2-way anova test results for feedback according to the origin brain area of the most active neuron and its average number of spikes.
The two-way anova test states that both variables including their interaction term are statistically significant because all of their p-values are lower than 0.05. This indicates that not only would changing the origin area and the average number of spikes may change the feedback type, but that the relationship between area and average number of spikes changes depending on which origin area the most active neuron is from.
Moving forward from analyzing the effect of brain area on feedback type, the contrasts in brain stimulus will be investigated.
Figure 2.18: Stacked barchart of successes and failures among each type of turn expectation.
| -1 | 1 | |
|---|---|---|
| left | 0.2547459 | 0.7452541 |
| random | 0.4952381 | 0.5047619 |
| right | 0.2446084 | 0.7553916 |
| still | 0.3428155 | 0.6571845 |
Figure 2.19: Proportion table displaying the success rates for each type of turn.
The correct type of turn is determined by the right and left contrasts. When right contrast exceeds left contrast, a left turn is expected. When left contrast exceeds right contrast, a right turn is expected. When the contrasts are both 0, the wheel is expected to be still. When the contrasts are equal and non-zero, the turn expectation is random. The chart demonstrates that the right turn is the most common turn expectation with a total of 1762 trials. The left turn is the second most common type of turn, being expected in 1633 trials. The turn expectation is still for 1371 trials. The random turn expectation is the least common, being expected in 315 trials. The right turn has the highest success rate of 75.5%. The left turn has a similar success rate of 74.5%. The success rate when the wheel is supposed to be still is 65.7% and and success rate for randome wheel turns is 50.5%.
Figure 2.20: Stacked barchart of successes and failures depending on turn expectation for mouse Cori.
| -1 | 1 | |
|---|---|---|
| left | 0.4170 | 0.5830 |
| random | 0.5306 | 0.4694 |
| right | 0.2516 | 0.7484 |
| still | 0.3373 | 0.6627 |
Figure 2.21: Proportion table for successes and failures depending on turn expectation for mouse Cori.
The left turn expectation is the most common correct turn expectation and the still, right, random turn expectations are the following most frequent turn expectations in that order. The random turn has the lowest success rate at 46.9% and the right expectation success rate is the highest at 74.8%. The right expectation rate is significantly higher than the left expectation success rate which is 58.3%. The success rate for the still wheel expectation is 66.2%, similar to the one of the general dataset.
Figure 2.22: Stacked barchart of successes and failures depending on turn expectation for mouse Forssmann
| -1 | 1 | |
|---|---|---|
| left | 0.2259 | 0.7741 |
| random | 0.4667 | 0.5333 |
| right | 0.4048 | 0.5952 |
| still | 0.2704 | 0.7296 |
Figure 2.23: Proportion table for successes and failures depending on turn expectation for mouse Forssmann*
The types of turn expectation in order of frequency are left, right, still, and random. The highest success rate is 77.4% for the left turn and the lowest success rate is 53.3% for the random expectation turn. The left success rate is significantly higher than the right success turn at 59.5%. The still expectation rate is approximately 73%, also significantly higher than the right success rate and higher than the one for the general dataset.
Figure 2.24: Stacked barchart of successes and failures depending on turn expectation for mouse Hench
| -1 | 1 | |
|---|---|---|
| left | 0.3475 | 0.6525 |
| random | 0.5957 | 0.4043 |
| right | 0.2680 | 0.7320 |
| still | 0.2788 | 0.7212 |
Figure 2.25: Proportion table for successes and failures depending on turn expectation for mouse Hench.
Above are the stacked barchart of successes and failures depending on turn expectation for mouse Hench and the associated proportion table. The ranking of turn expectation in frequency from highest to lowest are right, left, still, and random. The highest success rate if 73.2% for the right turn and the lowest is 40.4% for the random turn. The left turn success rate is significantly lower at 65.3% than the right turn success rate. The still expectation success rate is 72.1%, which is very similar to of that for mouse Forssmann and is considerably higher than the left success rate.
Figure 2.26: Stacked barchart of successes and failures depending on turn expectation for mouse Lederberg
| -1 | 1 | |
|---|---|---|
| left | 0.1542 | 0.8458 |
| random | 0.4021 | 0.5979 |
| right | 0.1470 | 0.8530 |
| still | 0.4164 | 0.5836 |
Figure 2.27: Proportion table for successes and failures depending on turn expectation for mouse Hench.
The ranking of turn expectation in frequency from highest to lowest are right, left, still, and random. The highest success rate is 85.3% for the right turn expectation and the lowest success rate is 58.4% for the still wheel expectation. The left turn success rate is similar to the right turn success rate at 84.6%. The success rate is 59.8% for the random turn expectation.
In conclusion, for mice Hench and Cori, performance was considerably better when a right turn was expected compared to when a left turn was. Mouse Forssman performed considerably better for when a left turn was expected versus when a right turn was. Mouse Lederberg is the only mouse who performed fairly equally on both left and right turn expectations, doing so with high success rates. The frequency of certain turn expectations do not appear to have an effect on the success rates. The success rates for random expectation were consistently low which is expected because the correct turn is randomly selected. The still wheel expectation was the 2nd highest success rate for each mouse except for Mouse Lederberg, where its success rate was considerably lower compared to other mice. Overall, it is highly suggested that the turn expectation has a relationship with the feedback type, with the specific relationship varying depending on the mouse.
Knowing that the turn expectation may be an important feature, the exact contrast between the left and right will be examined. Contrast difference will be calculated by subtracting the right contrast from the left. Therefore, a negative contrast difference will indicate an expected left turn and a positive difference will indicate an expect right turn. A difference of 0 indicates either a random turn or the expectation of a wheel to be still.
|
|
Figure 2.28: Proportion tables of contrast difference and feedback type, and of contrast difference and mouse.
It’s observed that among the failures and successes, there is a lower success rate of 62.87% for when the the contrast is not 0. This further supports the previous findings which indicated that among all sessions, mice had higher success rates for turning the wheel compared to keeping it still or randomly choosing. It is also noted that the absolute value of 0.25 for contrast different has a lower success rate at 68.9% and 64.35% for left and right turns, respectively. The success rates for when the absolute value of the contrast difference is 0.5 or above are all above 70%.
Each mouse appears to have a different distribution of contrast differences. A contrast difference of 0 appears to be the most popular contrast difference, but the most popular non-zero contrast difference varies for each mouse. Cori’s most popular difference is -1 at 14.33%, Forssmann’s is 0.25 at 10.05%, Hench’s is 1 at 13.96%, and Lederberg’s is 0.5 at 10.87%. In order to investigate whether contrast difference may have a relationship with the differing success rates for each mouse, a 2-way anova test will be performed.
## Df Sum Sq Mean Sq F value Pr(>F)
## contrast_diff 1 2 2.359 2.914 0.0879 .
## mouse_name 3 39 12.917 15.954 2.54e-10 ***
## contrast_diff:mouse_name 3 35 11.770 14.536 2.00e-09 ***
## Residuals 5073 4107 0.810
## ---
## Signif. codes: 0 '***' 0.001 '**' 0.01 '*' 0.05 '.' 0.1 ' ' 1Figure 2.29: 2-way anova test results of feedback according to contrast difference and mouse name.
A two-way anova test is performed to evaluate the relationship between mouse type and contrast difference when it comes to whether a mouse succeeds or fails. From the results, it’s shown that the p-value for mouse name and contrast difference are both significant since both are under 0.1. However, mouse name is much more significant since its p-value is so low. Additional, the interaction term between mouse name and contrast difference is very low, which indicates that the relationship between mouse and contrast difference varies depending on the type of mouse and that the interaction is statistically significant. In conclusion, both contrast difference and mouse name and the relationship between them are suggested to have an correlation with the feedback type.
Pivoting from the turn expectation, the average and total number of spikes will be investigated to determine whether they behave different among sessions and trials.
Figure 2.30: Violin plots for the number of spikes and average number of spikes per neuron for each trial, sorted based on mouse was performing and feedback type.
For both plots, the distributions for success and failure are relatively similar. However, among most of the plots, it is recognized that the point distribution for failures are more concentrated. This is seen when the plots’ widest points when feedback = -1 are wider than the widest points when feedback = 1. This indicates that the number of spikes or average number of spikes are a more telling variable for failures than successes. It is also observed that all of the plots for average number of spikes have a wider distribution when the feedback = 1. This indicates that successes have a more diverse range of average number of spikes. For the plots depicting number of spikes, the ranges only end higher when the feedback = 1. Therefore, the number of spikes may be more reliable than average number of spikes when estimating successful turns. When it comes to differences among the plots, the plots for average number of spikes are more heterogeneous according to feedback type than the ones for number of spikes, indicating that average number of spikes could be a more telling feature for determining feedback type. In conclusion, there are slightly differences between plots which further indicate different behaviors among mice and imply a relationship between feedback type and both the number of spikes and the average number of spikes.
It is evident that the activity of neurons from certain brain areas vary greatly over trials among each session. Different brain areas such as MRN have higher average spike counts than areas such as SCs in Session 14. Certain areas have relatively constant average spikes over time such as VISpm in Session 2 while area root in the same session appears to drop around 30 trials and stay relatively constant around 60 trials.
Figure 2.31: Graphs of the average spikes counts of each brain area in session 1, containing mouse Cori’s performance, for successes (feedback = 1) and failures (feedback = -1).
Looking at the graphs, the lines are more smooth for the success graph, but this is likely attributed to there being a higher sample size for successes than failures. Overall, it’s observed that the behavior among the averaged spike counts per brain area really does not vary much according to each different feedback type.
Figure 2.32: Graphs for the average spike counts for the brain areas in session 17, for mouse Lederberg, divided into successes and failures.
Looking at the same graphs for mouse Lederberg, some slight differences between the behaviors for different feedback types are observed. For example, the general trend line for LD appears to decrease slightly over time when feedback = 1 and slightly increases over time when feedback = -1. However, the behaviors for each brain area are still similar to each other despite different feedback. The LD brain area even has the same spikes around 200 trials in both graphs. It can be concluded from the graphs on the two randomly chosen sessions from the performances of difference mice that the average spike counts of the brain areas may be homogenous across trials. Therefore, it may not be a telling predictor of the feedback type.
Figure 2.33: Graphs of the neural activity of different brain areas in trial 114 of session 1 for both a successful and non-successful trial.
Taking the first and last trials of session 1, we can see the neural patterns which led to a successful and an unsuccessful turn. It is observed that order of the distribution definitely varies more than the behavior of the average spikes of brain areas because of the different scattering of the points from different brain areas.
Looking at the differences between the averaged timebins of successes and failures according to different turn expectations, it is observed that the left and right turn have similar neural patterns while random turn and still turn have different ones. This indicates that the neural behavior for left and right turns may be similar. This could also indicate that 3 models may be needed for building a final prediction model because the neural patterns differ.
Section 3: Data Integration
To integrate all the data into one dataset, several factors were considered to best consolidate the data. Several variables were removed and added in this process. Basic and essential information such as the mouse name, session number, trial number, and feedback type were kept. The number of brain areas was not included because it did not appear to have a correlation with the success rate, and was unique to each session. The date was not kept because it was unique to each session and the session number is already logged.
While contrast difference proved to be a statistically significant feature, it was not included in data integration because it would be perfectly collinear with left and right contrast, which were prioritized over the difference because they provided more detail. The turn expectation variable was added which is a categorical variable keeping track of the correct expected turn for each trial. The brain origins of the 3 most active neurons for each trial were not included in this integration because while they proved to be heterogeneous across sessions and trials, there were several regions that only occurred a few times. This would cause major issues in model prediction due to unseen levels and variables. However, the associated average spikes for each neuron are included because it proved to be statistically significant to the feedback type in the 2-way anova test and is easier for models to work with.
Knowing that the individual neural activity for each trial was heterogeneous, the firing rates of neurons are an important feature to incorporate in the dataset. However, due to the inconsistent dimensions for each trial and the sheer scale, the matrices in the spks list have been averaged to the average number of spikes per time bin, resulting in 40 variables since there is a column for the average of each of the 40 time bins in a trial. Because the average number of spikes did not appear to be very heterogeneous across sessions, it was not kept. Additionally, it would have been collinear with the averaged time bins, providing further reason to leave it out. Additionally, mouse name and turn expectation were one hot encoded so that models such as ridge regression could be used.
| session_num | trial_n | contrast_left | contrast_right | feedback | n_spikes | n_neurons | avg_spike_area1 | avg_spike_area2 | avg_spike_area3 | TB1 | TB2 | TB3 | TB4 | TB5 | TB6 | TB7 | TB8 | TB9 | TB10 | TB11 | TB12 | TB13 | TB14 | TB15 | TB16 | TB17 | TB18 | TB19 | TB20 | TB21 | TB22 | TB23 | TB24 | TB25 | TB26 | TB27 | TB28 | TB29 | TB30 | TB31 | TB32 | TB33 | TB34 | TB35 | TB36 | TB37 | TB38 | TB39 | TB40 | mouseCori | mouseForssmann | mouseHench | mouseLederberg | turn_expleft | turn_exprandom | turn_expright | turn_expstill |
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| 1 | 1 | 0.0 | 0.50 | 1 | 1161 | 734 | 0.600 | 0.425 | 0.425 | 0.0490463 | 0.0367847 | 0.0177112 | 0.0149864 | 0.0326975 | 0.0286104 | 0.0313351 | 0.0122616 | 0.0340599 | 0.0190736 | 0.0463215 | 0.0204360 | 0.0326975 | 0.0613079 | 0.0422343 | 0.0422343 | 0.0272480 | 0.0190736 | 0.0367847 | 0.0326975 | 0.0517711 | 0.0490463 | 0.0490463 | 0.0504087 | 0.0435967 | 0.0653951 | 0.0177112 | 0.0422343 | 0.0449591 | 0.0476839 | 0.0626703 | 0.0490463 | 0.0572207 | 0.0613079 | 0.0313351 | 0.0367847 | 0.0504087 | 0.0544959 | 0.0395095 | 0.0395095 | 1 | 0 | 0 | 0 | 1 | 0 | 0 | 0 |
| 1 | 2 | 0.0 | 0.00 | 1 | 963 | 734 | 0.625 | 0.525 | 0.425 | 0.0299728 | 0.0313351 | 0.0340599 | 0.0272480 | 0.0258856 | 0.0313351 | 0.0217984 | 0.0231608 | 0.0231608 | 0.0340599 | 0.0272480 | 0.0272480 | 0.0245232 | 0.0367847 | 0.0476839 | 0.0381471 | 0.0217984 | 0.0149864 | 0.0395095 | 0.0326975 | 0.0395095 | 0.0435967 | 0.0272480 | 0.0272480 | 0.0326975 | 0.0190736 | 0.0299728 | 0.0395095 | 0.0258856 | 0.0217984 | 0.0245232 | 0.0395095 | 0.0558583 | 0.0435967 | 0.0422343 | 0.0531335 | 0.0613079 | 0.0381471 | 0.0286104 | 0.0258856 | 1 | 0 | 0 | 0 | 0 | 0 | 0 | 1 |
| 1 | 3 | 0.5 | 1.00 | 0 | 1354 | 734 | 0.775 | 0.550 | 0.525 | 0.0490463 | 0.0504087 | 0.0299728 | 0.0435967 | 0.0245232 | 0.0408719 | 0.0299728 | 0.0381471 | 0.0340599 | 0.0422343 | 0.0558583 | 0.0408719 | 0.0490463 | 0.0504087 | 0.0381471 | 0.0463215 | 0.0435967 | 0.0667575 | 0.0653951 | 0.0626703 | 0.0558583 | 0.0504087 | 0.0449591 | 0.0585831 | 0.0681199 | 0.0544959 | 0.0667575 | 0.0544959 | 0.0667575 | 0.0585831 | 0.0653951 | 0.0367847 | 0.0340599 | 0.0231608 | 0.0299728 | 0.0422343 | 0.0354223 | 0.0272480 | 0.0381471 | 0.0313351 | 1 | 0 | 0 | 0 | 1 | 0 | 0 | 0 |
| 1 | 4 | 0.0 | 0.00 | 0 | 1014 | 734 | 0.750 | 0.625 | 0.525 | 0.0558583 | 0.0531335 | 0.0272480 | 0.0613079 | 0.0572207 | 0.0599455 | 0.0449591 | 0.0286104 | 0.0395095 | 0.0354223 | 0.0367847 | 0.0286104 | 0.0408719 | 0.0354223 | 0.0381471 | 0.0490463 | 0.0272480 | 0.0190736 | 0.0286104 | 0.0258856 | 0.0177112 | 0.0108992 | 0.0313351 | 0.0354223 | 0.0299728 | 0.0245232 | 0.0190736 | 0.0258856 | 0.0408719 | 0.0204360 | 0.0272480 | 0.0163488 | 0.0395095 | 0.0231608 | 0.0435967 | 0.0381471 | 0.0272480 | 0.0340599 | 0.0381471 | 0.0449591 | 1 | 0 | 0 | 0 | 0 | 0 | 0 | 1 |
| 1 | 5 | 0.0 | 0.00 | 0 | 1046 | 734 | 0.925 | 0.575 | 0.475 | 0.0272480 | 0.0435967 | 0.0313351 | 0.0245232 | 0.0449591 | 0.0381471 | 0.0463215 | 0.0572207 | 0.0476839 | 0.0163488 | 0.0272480 | 0.0177112 | 0.0408719 | 0.0299728 | 0.0463215 | 0.0313351 | 0.0476839 | 0.0313351 | 0.0544959 | 0.0299728 | 0.0422343 | 0.0435967 | 0.0395095 | 0.0258856 | 0.0381471 | 0.0231608 | 0.0354223 | 0.0245232 | 0.0217984 | 0.0422343 | 0.0422343 | 0.0340599 | 0.0258856 | 0.0395095 | 0.0395095 | 0.0476839 | 0.0354223 | 0.0286104 | 0.0286104 | 0.0326975 | 1 | 0 | 0 | 0 | 0 | 0 | 0 | 1 |
| 1 | 6 | 0.0 | 0.00 | 1 | 803 | 734 | 0.875 | 0.650 | 0.625 | 0.0490463 | 0.0217984 | 0.0163488 | 0.0108992 | 0.0122616 | 0.0231608 | 0.0272480 | 0.0326975 | 0.0163488 | 0.0190736 | 0.0299728 | 0.0190736 | 0.0326975 | 0.0245232 | 0.0231608 | 0.0313351 | 0.0217984 | 0.0272480 | 0.0313351 | 0.0190736 | 0.0258856 | 0.0190736 | 0.0272480 | 0.0217984 | 0.0272480 | 0.0258856 | 0.0435967 | 0.0599455 | 0.0435967 | 0.0476839 | 0.0395095 | 0.0531335 | 0.0163488 | 0.0340599 | 0.0258856 | 0.0177112 | 0.0217984 | 0.0149864 | 0.0204360 | 0.0190736 | 1 | 0 | 0 | 0 | 0 | 0 | 0 | 1 |
| 1 | 7 | 1.0 | 0.50 | 1 | 1543 | 734 | 0.600 | 0.575 | 0.575 | 0.0544959 | 0.0367847 | 0.0367847 | 0.0381471 | 0.0490463 | 0.0340599 | 0.0163488 | 0.0231608 | 0.0381471 | 0.0190736 | 0.0381471 | 0.0340599 | 0.0572207 | 0.0408719 | 0.0449591 | 0.0640327 | 0.0449591 | 0.0667575 | 0.0558583 | 0.0504087 | 0.0476839 | 0.0558583 | 0.0599455 | 0.0585831 | 0.0831063 | 0.0722071 | 0.0653951 | 0.0803815 | 0.1144414 | 0.0694823 | 0.0435967 | 0.0504087 | 0.0817439 | 0.0831063 | 0.0531335 | 0.0504087 | 0.0476839 | 0.0449591 | 0.0395095 | 0.0572207 | 1 | 0 | 0 | 0 | 0 | 0 | 1 | 0 |
| 1 | 8 | 0.5 | 0.00 | 1 | 1251 | 734 | 0.650 | 0.625 | 0.475 | 0.0613079 | 0.0231608 | 0.0326975 | 0.0258856 | 0.0326975 | 0.0381471 | 0.0490463 | 0.0544959 | 0.0463215 | 0.0367847 | 0.0326975 | 0.0272480 | 0.0286104 | 0.0299728 | 0.0517711 | 0.0286104 | 0.0626703 | 0.0490463 | 0.0517711 | 0.0340599 | 0.0313351 | 0.0476839 | 0.0490463 | 0.0422343 | 0.0313351 | 0.0381471 | 0.0286104 | 0.0326975 | 0.0490463 | 0.0422343 | 0.0313351 | 0.0354223 | 0.0299728 | 0.0463215 | 0.0667575 | 0.0626703 | 0.0572207 | 0.0667575 | 0.0844687 | 0.0340599 | 1 | 0 | 0 | 0 | 0 | 0 | 1 | 0 |
| 1 | 9 | 0.0 | 0.00 | 1 | 1108 | 734 | 1.350 | 0.550 | 0.450 | 0.0286104 | 0.0217984 | 0.0313351 | 0.0204360 | 0.0367847 | 0.0245232 | 0.0504087 | 0.0504087 | 0.0395095 | 0.0231608 | 0.0408719 | 0.0326975 | 0.0435967 | 0.0367847 | 0.0258856 | 0.0367847 | 0.0476839 | 0.0381471 | 0.0572207 | 0.0449591 | 0.0449591 | 0.0326975 | 0.0313351 | 0.0463215 | 0.0435967 | 0.0286104 | 0.0435967 | 0.0463215 | 0.0258856 | 0.0354223 | 0.0463215 | 0.0163488 | 0.0326975 | 0.0340599 | 0.0313351 | 0.0367847 | 0.0476839 | 0.0422343 | 0.0476839 | 0.0640327 | 1 | 0 | 0 | 0 | 0 | 0 | 0 | 1 |
| 1 | 10 | 0.5 | 0.25 | 1 | 1271 | 734 | 0.975 | 0.550 | 0.475 | 0.0217984 | 0.0313351 | 0.0299728 | 0.0326975 | 0.0367847 | 0.0395095 | 0.0490463 | 0.0326975 | 0.0326975 | 0.0408719 | 0.0367847 | 0.0313351 | 0.0531335 | 0.0408719 | 0.0572207 | 0.0504087 | 0.0558583 | 0.0708447 | 0.0531335 | 0.0408719 | 0.0544959 | 0.0490463 | 0.0517711 | 0.0681199 | 0.0544959 | 0.0599455 | 0.0435967 | 0.0463215 | 0.0572207 | 0.0667575 | 0.0354223 | 0.0435967 | 0.0408719 | 0.0231608 | 0.0408719 | 0.0340599 | 0.0258856 | 0.0395095 | 0.0299728 | 0.0286104 | 1 | 0 | 0 | 0 | 0 | 0 | 1 | 0 |
Section 4: Predictive Modeling
To build a prediction model to predict the feedback type, 3 datasets are created and split into test and training datasets to ultimately select 3 models for prediction. As noted in EDA, the difference between the averaged time bins had different patterns according to type of correct turn expectation. While it was also noted that the behavior of mice varied, separate models were not created for each mouse to preserve sample size and because the mouse name is still a recorded variable. The left and right turn expectations had similar patterns while the random turn and still expectations were different. Therefore, 3 models will be selected for predicting the feedback type of trials where a left or right turn is expected, when the wheel is expected to be still, and for random turn expectations. For the left/right dataset, all of the variables in the integrated data set are kept. For the still wheel dataset, the turn expectation was removed because it was homogeneous, and the left and right contrasts were removed because they both are constantly 0. For the random turn dataset, turn expectation is removed because it is always “random” and the right contrast was removed because the left and right contrast are equal, so only one is needed. 80% of each dataset is kept for training and the remaining 20% is kept for testing.
A simple logistic regression model is fit on each dataset. For the left/right dataset, the training accuracy is 78.13% and the testing accuracy is 77.9%. The test accuracy is approximately 2% higher than the overall success rate for left and right turns, 75.03%. The log regression model for the still dataset did not perform as well. The training accuracy is 68.64% but the testing accuracy is 61.31%, which is lower than 65.72%, the overall success rate for still wheel expectations. The random dataset logistic model overfit with a training accuracy os 70.63% and a testing accuracy of 49.21%, slightly lower than 50.48%, the overall success rate for random turn expectation. However, this is expected since the correct wheel turn is randomly chosen, so it is indicated that no variable can properly predict whether the mouse can turn the wheel successfully.
Expanding on logistic regression, ridge regression models are tested due to the amount of collinear coefficients and to fight overfitting seen in the still and random turn expectations. For the left/right dataset, the train accuracy is 75.68% and the test accuracy is 76.92%, which is marginally better than the test accuracy for the logistic regression model for the left/right dataset. For the still wheel dataset, the training accuracy is 66.75% and the testing accuracy is 62.69%. This model overfit less and the testing accuracy is better than the one from the logistic regression model. However, the testing accuracy is still approximately 3% lower than the naive success rate for still wheel turns. For the random dataset, the training accuracy is 61% and the test accuracy is 50.95%. This model still overfits but not to a less extent than the one from simple logistic regression.
Moving forward, random forest models prove to have higher accuracies but also tend to overfit all the data sets. All of the training accuracies were 100%. The left/right model test accuracy is 78.65%, the still wheel model test accuracy is 63.87%, and the random turn model test accuracy is 49.20% All of these training accuracies are higher than the previous models, even on the random turn model. While the still wheel model test accuracy is still slightly below the success rate, it is getting closer. To address overfitting and optimize accuracy, 10-fold cross validation is performed in combination with the random forest classifier to fine tune the hyper parameters. The test accuracy of the left/right model increased to 80.85%, but the test accuracy of both the still dataset and random dataset decreased to 62.77% and 55.56%, respectively. It is noted that the test accuracy of the still dataset dipped below the general success rate for still wheel turn expectations. Because the random forests models performed the best on both the left/right and still datasets, the final models will either be selected from the base random forest models or the cross validation ones with the except of the model for the random dataset. The final model for the random turn dataset will be the ridge regression model because it overfit the least.
From the confusion matrices, it is observed that the 10-fold classifier misclassified successes more but also increased the accuracy for predicting failures. For the first random forest classifier, the majority of failures were classified as successes while the 10-fold classifier accurately predicts almost double the amount in the first one.
The ROC curves confirm that the 10-fold classifier left/right model is better at distinguishing between the classes because the AUC is approximately 0.65 larger than the AUC of 0.6 for the base model. Therefore, the 10-fold classifier will be selected for predicting trials with a left or right turn as the expected correct turn.
The two models for the the still dataset perform very similarly. The 10-fold model correctly classifiers 2 more succeses than the base model, but majorly overclassifies the failure class as success due to the class imbalance. The base model has a higher accuracy for the failure class for a marginally lower accuracy in the success class, so the base model for the still dataset will be selected.
The ROC curves demonstrate that the base model for the still dataset has a higher AUC of 0.56 compared to the 10-fold model’s AUC of 0.5. Therefore, the base model can better distinguish the classes and will be used as the final prediction model for predicting the feedback type of trials with a still wheel turn expectation as the correct response.
In conclusion, after testing logistic regression, ridge logistic regression, random forest classifiers, and 10-fold random forest classifiers, it was deduced that the 10-fold random forest classifiers performed the best on the left/right dataset while the base random forest model performed the best on the still dataset and was chosen for the random dataset
Section 5: Prediction Performance on the Test Sets
After splitting the test data into 3 subsets of data of the trials with left/right turn expectations, still turn expectations, and random turn expectations, the respective success rates are 76.81%, 66.04%, and 0.44%. When testing each respective data set on the predictive models, the test accuracy for left/right is 70.23%, the test accuracy for still turns is 66.04%, and the test accuracy for random turns is 56.89%. While the test accuracy for the left/right dataset is approximately 6% lower than the success rate, it is still fairly close. Because the sample size of this data is smaller because it was further split into 3 sets, it is expected to have more variance in the accuracy. The test accuracy for still wheel expectation is incredibly close to the success rate at 66.04%, indicating fair performance. The random model success rate is approximately 6.9% higher than the expected success rate but this could also be due to the variance due to small sample size. In conclusion, the left/right model performed slightly worse than it did during model building but the still wheel and random turn expectation models performed at equal or better performance.
Section 6: Discussion
This project consisted of data analysis and wrangling to reveal the features which would help predict the feedback type from the information provided in a mouse’s trial. It was noted that there was great heterogeneity across sessions and trials, but also homogeneity which was noticed while noticing the averaged neural spike patterns of the different correct turn expectations. Splitting the data set into 3 subsets to create 3 models depending on the turn expectation proved to be helpful for being able to select the best model for each type of turn, but also resulted in models that did not perform as well as expected. By exploring the neural behavior of each turn expectation, I had thought that making 3 different models would result in better accuracy for each type of expected turn, but the models performed relatively closely to the original success rates. However, these models were ultimately chosen for the final prediction model because the test accuracy especially for the left/right dataset during model building was considerably higher than the original success rate. This accuracy, however, dropped for the new test data. However, this could be because of the smaller sample size after dividing the already smaller dataset into 3 datasets. In the future, I would experiment more with feature engineering because so many categorical variables were one-hot encoded, which may have contributed a lot of the noise in the dataset. I would try label encoding or try more models with a lot less features. Additionally, I would also experiment more with creating one model which encompasses all of the turn expectations. Overall, a very in-depth exploratory data analysis was performed on the behavior which led to success and unsuccessful decisions, ultimately leading to building a prediction model consisting of 3 models to account for the heterogeneity across trials.
Acknowledgements
ChatGPT: https://chat.openai.com/share/8d5fc77c-264f-4e1b-aaed-3da33e8721e4
Steinmetz, N.A., Zatka-Haas, P., Carandini, M. et al. Distributed coding of choice, action and engagement across the mouse brain. Nature 576, 266–273 (2019). https://doi.org/10.1038/s41586-019-1787-x